
# 2019 February Update    
 Welcome to the first progress post! These posts should be a summary in the development and progress of Janis    
and its related projects. You can find more information on the 
[documentation](https://janis.readthedocs.io/en/latest/).    
    
This has been a portable month, a lot of effort has been placed into improving the 
portability and reproducibility of workflows.  
  
  
## Janis and Project Restructure  
  
The project has a more permanent name: **Janis**.  
  
In February the project restructure was completed, and a lot of automatic builds and releases were built.   
All of the projects are on Pip (see the [README](https://github.com/PMCC-BioinformaticsCore/janis) 
for more information).  


|  | build  | docs  | pypi | codecov |
|---|:-:|:-:|:-:|:-:|
| Janis |  [![Build Status](https://travis-ci.org/PMCC-BioinformaticsCore/janis.svg?branch=master)](https://travis-ci.org/PMCC-BioinformaticsCore/janis)  | [![Documentation Status](https://readthedocs.org/projects/janis/badge/?version=latest)](https://janis.readthedocs.io/en/latest/?badge=latest) | [![PyPI version](https://badge.fury.io/py/janis-pipelines.svg)](https://badge.fury.io/py/janis-pipelines) |  [![codecov](https://codecov.io/gh/PMCC-BioinformaticsCore/janis/branch/master/graph/badge.svg)](https://codecov.io/gh/PMCC-BioinformaticsCore/janis) |
| Bioinformatics |[![Build Status](https://travis-ci.org/PMCC-BioinformaticsCore/janis-bioinformatics.svg?branch=master)](https://travis-ci.org/PMCC-BioinformaticsCore/janis-bioinformatics) | See Janis  |  [![PyPI version](https://badge.fury.io/py/janis-pipelines.bioinformatics.svg)](https://badge.fury.io/py/janis-pipelines.bioinformatics)  |   |
| Shepherd | | | | |
| CWL-Gen | [![Build Status](https://travis-ci.org/illusional/python-cwlgen.svg?branch=master)](https://travis-ci.org/common-workflow-language/python-cwlgen) |   |[![PyPI version](https://badge.fury.io/py/illusional.cwlgen.svg)](https://badge.fury.io/py/illusional.cwlgen) | [![codecov](https://codecov.io/gh/illusional/python-cwlgen/branch/master/graph/badge.svg)](https://codecov.io/gh/illusional/python-cwlgen)|
| WDL-Gen | [![Build Status](https://travis-ci.org/illusional/python-wdlgen.svg?branch=master)](https://travis-ci.org/illusional/python-wdlgen) || [![PyPI version](https://badge.fury.io/py/illusional.wdlgen.svg)](https://badge.fury.io/py/illusional.wdlgen) | |

You can install all of these by:
```bash
pip3 install janis-pipelines[bioinformatics]
```
And then you can construct a simple workflow through:
```python  
import janis as j  
from janis.unix.tools.echo import Echo

w = j.Workflow("workflow_identifier")  
  
inp = j.Input("input_identifier", j.String())  
step = j.Step("step_identifier", Echo())  
outp = j.Output("output_identifier")  
  
w.add_pipe(inp, step, outp)  
  
# Will print the CWL, input file and relevant tools to the console  
w.translate("cwl")  
w.translate("wdl")  
```
    
    
## Portability and reproducibility    
 One of the goals of the project is to support multiple backends without a lot of configuration.     
Likely the most effective method to promote this flexibility is to wrap each tool in a container technology.     
By far, Docker is the most popular and widely supported container technology, however due to the Docker    
    
As part of these changes, I've followed a couple of merge requests ([merge here]) and managed to start a stronger    
conversation about using different container technologies with Cromwell simply by building a config file.    
    
As part of this conversation, we started a container tutorial:    
- Pull Request: https://github.com/broadinstitute/cromwell/issues/2177    
- Current tutorial: https://github.com/illusional/cromwell/blob/develop/docs/tutorials/Containers.md    
    
Basically, by creating a new config backend, we can override the `submit-docker` block and provide a     
`singularity exec` or `udocker run` command:    

```hocon 
include required(classpath("application"))    

backend {
  default: singularity
  providers: {
    singularity {
      # The backend custom configuration.
      actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"

      config {
        run-in-background = true
        runtime-attributes = """
         String? docker
        """
        submit-docker = """
         singularity exec --bind ${cwd}:${docker_cwd} docker://${docker} ${job_shell} ${script}
        """
      }
    }
  }
}
```    

 For a job scheduler, we introduce a few more constraints     
- Make sure Singularity is loaded (and in our path)    
- We should treat worker nodes as if they do not have stable access to the internet or build access,     
so we will pull the container before execution of the task.    
- It's a good idea to ask Singularity to build the image into the execution directory of the task as     
an artifact, and to save rendering time on the worker node.    
- OR: Let's only build the container once for every execution of that container.    
    
    
### Issues    

- udocker is _very_ slow to run containers. This is because it builds the container for each `run`.     
- Singularity doesn't really offer a way to ensure a container is built. You can build to a directory 
and it will prompt the user to override. There has been a request to submit a feedback on this
 [here](https://github.com/broadinstitute/cromwell/pull/4635#discussion_r258743451).    
    
  > Ideally we want Singularity to work like Docker, which I think can handle multiple simultaneous pull commands.     
    [Michael Milton] opened a Cromwell issue [here](https://github.com/broadinstitute/cromwell/issues/4673) that would     
    solve it from one angle, but maybe this problem is better solved by better image caching in Singularity?    


### Notes    

- By not including the `submit` block, we enforce the requirement that every task should have a container.    
    
- `udocker` needs to include `docker.hash-lookup.enabled = false` at the top level of the configuration file, 
as `udocker` does [not currently support](https://github.com/indigo-dc/udocker/issues/112) the use 
[docker digests](https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier).    
    
    
## Amazon + AWS    
 Now that we have some stability in the produced result, and I have a bit more experience writing 
 configuration files, it was time to have another attempt at running the pipeline on AWS. 
 I've setup the AWS Cromwell stack in two different ways to test the workflows.  
  
Basically, I've been trying to run a super simple workflow that requests a file from S3, and `cat` 
the results to stdout, and been having some troubles doing this with both WDL and CWL.  
  
### WDL  
  When running the WDL workflow, it successfully submits the job to AWSBatch 
  (`id: e39d117d-6319-4eae-a00e-7a5f1633bb1d`), and succeeds:  
```  
Workflow test complete. Final Outputs:  
{  
  "test.cat_string.response": "s3://cromwell-logs/cromwell-execution/test/5006438c-d14c-4eb0-9034-3a2ac5129311/call-cat_string/cat_string-stdout.log"
}  
```  
  
### CWL  
  
This is where the problems seem to be. I've tried this in two ways, first by 
including the batch config, and one that simply turns on the S3 files system (no-batch).   
  
With the batch system turned on, the workflow fails with the following error:  
```  
Call input and runtime attributes evaluation failed for cwl_temp_file_aac32f48-4b1b-471a-b1ff-577febf99390.cwl:  
Failed to evaluate input 'name' (reason 1 of 1): [Attempted 1 time(s)] - S3Exception: Access Denied (Service: S3Client; Status Code: 403; Request ID: 5B2CC9584E31F57E)  
```  
  
Similar, with the batch system turned off (just the S3 filesystem turned on), it fails with the same error:  
```  
Call input and runtime attributes evaluation failed for cwl_temp_file_f84ebdb9-bce1-4c27-b3c1-9701a977176b.cwl:  
  
Failed to evaluate input 'name' (reason 1 of 1): [Attempted 1 time(s)] - S3Exception: Access Denied (Service: S3Client; Status Code: 403; Request ID: 407699495723E959)  
```  
  
I've also run the workflow with a localised file instead of in S3 - this successfully builds the job, 
submits to batch and then reasonably fails with:  
```  
cat: /home/ec2-user/hello.txt: No such file or directory  
```  
(This sounds more of a Cromwell issue, which I'll raise with them soon).  
  
I've ensured that I have the correct IAM role attached to the EC2 instance, and can access files 
through the CLI within the EC2 instance.  
  
I've done a little bit of searching and found https://github.com/broadinstitute/cromwell/issues/4586 
which details how Cromwell can fail to localise input files with no real resolution. But it's not 
exactly the same as my issue, so I'm not 100% sure.  
  
### Whole Genome Pipeline  
  
I've got this workflow as functional WDL and CWL. Instead of hosting the human reference genome, 
we're using [Broad's HG38](https://s3.amazonaws.com/broad-references/broad-references-readme.html)
([OpenData page](https://registry.opendata.aws/broad-references/)): `s3://broad-references/hg38/v0/`.  
  
I'm able to poll and copy data from this repo within the EC2 instance, 
even though it's out of my region (`us-east-1` to `ap-southeast-2`).  
  
When running the workflow with CWL, I get the following failure:  
```  
Workflow input processing failed:  
[Attempted 1 time(s)] - S3Exception: null (Service: S3Client; Status Code: 301; Request ID: null)  
[Attempted 1 time(s)] - S3Exception: null (Service: S3Client; Status Code: 301; Request ID: null)  
[Attempted 1 time(s)] - S3Exception: null (Service: S3Client; Status Code: 301; Request ID: null)  
[Attempted 1 time(s)] - S3Exception: null (Service: S3Client; Status Code: 301; Request ID: null)  
[Attempted 1 time(s)] - S3Exception: null (Service: S3Client; Status Code: 301; Request ID: null)  
at cromwell.engine.workflow.lifecycle.materialization.MaterializeWorkflowDescriptorActor.cromwell$engine$workflow$lifecycle$materialization$MaterializeWorkflowDescriptorActor$$workflowInitializationFailed(MaterializeWorkflowDescriptorActor.scala:217)  
```  
I'm referencing the broad data set 5 times, so this makes sense to me, and the `301` to me sounds 
like it's an out-of-region issue or Cromwell isn't redirecting to the full `s3://` uri properly. 
I've tried to fully qualify the broad reference with the 
[full link](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro), 
(something like:  `http://s3-us-east-1.amazonaws.com/broad-references/hg38/v0/` but with the `s3://` 
prefix with no luck.  

When running the workflow with WDL, I get the following failure:  
```  
cromwell.engine.io.IoAttempts$EnhancedCromwellIoException: [Attempted 1 time(s)] - S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 7D2B88CEC3FCF844)  
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 7D2B88CEC3FCF844)  
```  
  
Which to me, maybe sounds like it can't write to the execution directory on the S3 bucket?  
  
___  
  
We've contacted AWS Support, and trying to ascertain more information about running on this target.  
  
  
## Toil  
  
I've investigated the use of the Toil Workflow Engine. Unfortunately it still 
[only works on Python 2.7](https://toil.readthedocs.io/en/latest/gettingStarted/install.html#preparing-your-python-runtime-environment):

There seemed to be some work going into making it compatible with Python 3+ 
([#1780](https://github.com/DataBiosphere/toil/issues/1780)) and the 
[Toil roadmap](https://github.com/DataBiosphere/toil/wiki/Roadmap) has this feature request 
as _Near Term (In progress)_. I've pinged some people on #1780, and awaiting more progress on this.

I've got it working with `pipenv`, and although small workflows succeed, there are issues with 
docker such as exporting of `/var` that prevent it from working. 
    
    
## Conversion to WDL  
  
After I ran the simple workflow on Amazon, it seemed that there are 
[issues](https://github.com/broadinstitute/cromwell/issues/4586) in the way that Cromwell 
localises files for CWL. As the goal of the project was to be language agnostic, 
it was a good time to revisit the WDL generation and increase the quality of the translation.  
  
The two big issues I had were:  
1. Directories in WDL  
2. Handling secondary files  
  
The first can be solved by using the
 [current WDL development spec](https://github.com/openwdl/wdl/blob/master/versions/development/SPEC.md) 
 of WDL (with Cromwell >= 37), the second is a bit harder.  
  
### Secondary Files  
  
The main issues that I had with the conversion was around secondary files, ensuring that 
files are passed around, mapped correctly and are picked back up by generated globs from 
the syntax language that CWL uses (as it's the same representation that Janis uses).  

> Note: this might be worth changing to be fully qualified for increased transparency, 
and then we can use base name everywhere, however this will complicate the CWL translation.  
  
This issue was spawned from:  and also within the 
[WDL forum](https://gatkforums.broadinstitute.org/wdl/discussion/9299/secondary-index-files-and-directories-in-wdl) 
where there was general support for accessory files.  
  
As part of this complication and other discussions 
([broadinstitute/cromwell#2269](https://github.com/broadinstitute/cromwell/issues/2269) 
| [WDL forum](https://gatkforums.broadinstitute.org/wdl/discussion/9299/secondary-index-files-and-directories-in-wdl)), 
I opened an issue on Github about [File Bundles and Secondary / Accessory Files](https://github.com/openwdl/wdl/issues/289).   
  
The current proposal would look something like this:  
```wdl  
Struct indexedBamFile {  
  File bam 
  Inferred { 
    File index = bam + ".bai"               # $name.bam.bai 
    File txt = sub(bam, "\\.bam$", ".txt")  # $name.txt    
 }  
}  
```  
  
In the mean time, I had to do a bit of tricky manipulation to get the WDL to generate properly. 
Basically, you can't bundle files together, so I took some inspiration from the 
[GATK Best Practices](https://github.com/gatk-workflows/gatk4-somatic-snvs-indels/blob/master/mutect2-normal-normal.wdl#L28) 
to use `filename_idx` as a guide, hence a `FastaWithDict` (with secondaries: `.amb`, `.ann`, `.bwt`, `.pac`, `.sa`, `.fai`, `^.dict`) becomes:  
```  
workflow whole_genome_germline {  
  input { 
    File reference 
    File reference_amb 
    File reference_ann 
    File reference_bwt 
    File reference_pac 
    File reference_sa 
    File reference_fai 
    File reference_dict 
  } 
  ...
```  
  
We can easily generate direct connections, just like `ref_task=reference,ref_task_amb=reference_amb, ...etc`.  
If you have an array of index files, we map them individually, eg:  
```wdl  
Array[File] inp2  
Array[File] inp2_txt  
```  
  
Now we can handle the case where you scatter based on the field with the secondary files, 
however this is a little more difficult. Basically you collect all the secondaries into one array, 
transform the array and then array-index them, eg:  
```wdl  
workflow scattered_test_workflow {  
  input { 
    Array[File] inp2 
    Array[File] inp2_txt 
  } 
  scatter (i in transform([inp2, inp2_txt])) { 
    call toolthatacceptsandreturnssecondary as stp2 {
      input: input=i[0], input_txt=i[1] 
    } 
  } 
  ...
}  
```  
  
Lastly we handle the glob output case, and this is where we have some issues. The 
[CWL Spec](https://www.commonwl.org/v1.0/CommandLineTool.html#CommandInputParameter) uses the query syntax:  
> 1. If string begins with one or more caret ^ characters, for each caret, remove the last 
file extension from the path (the last period . and all following characters). If there are no 
file extensions, the path is unchanged.  
> 2. Append the remainder of the string to the end of the file path.  
  
A Python implementation we use is as follows:  

```python  
def apply_secondary_file_format_to_filename(filename: Optional[str], secondary_file: str):    
    if not filename: return None    
    
    fixed_sec = secondary_file.lstrip("^")    
    leading = len(secondary_file) - len(fixed_sec)    
    if leading <= 0:    
        return filename + fixed_sec    
    
    split = filename.split(".")    
    return ".".join(split[:-min(leading, len(split) - 1)]) + fixed_sec  
```  
  
However, we don't really have access to the filename in a scope where we can split it or use more complex 
indexing. We handle this as follows:  
1. If there's no `^`, we just append the secondary extension, eg: `File output_txt = input + ".txt"`  
2. If we have a `Filename`, we know the extension so we can use the WDL function: 
[`sub`](https://github.com/openwdl/wdl/blob/master/versions/development/SPEC.md#string-substring-string-string)  
3. If the output is a file, we use the 
[`basename`](https://github.com/openwdl/wdl/blob/master/versions/development/SPEC.md#string-basenamestring) 
function, which allows us to pass an extension to remove off the end. However, it kind of has the caveat 
that it will remove all extensions and doesn't respect the number of `^`.   
  
### Shepherd

[Shepherd](https://github.com/PMCC-BioinformaticsCore/shepherd) is (going to be) a workflow runner, 
and a bit of a helper to avoid knowing how specific engines work. It's designed to encapsulate some 
of the environment logic, specifically for Peter Mac, Melbourne Uni (Spartan) and WEHI to allow 
running workflows without needing to construct configuration files.

A secondary use for Shepherd will be a testing suite. You can define tests with a tool, 
some inputs (+ params) and the expected output and it could theoretically run the workflow 
and compare the results. It should be able to submit jobs to the engine and workflow manager 
(Slurm, Torque, etc) and if all the passes succeed, you should be confident that the workflows 
that Janis generates will give reproducible results on the intended environment.

